Test Discovery

Given that we have written a number of Test Methods (page X) on one or more Testcase Classes (page X), we need to give the Test Runner (page X) some way to find the tests. Test Discovery eliminates most of the hassles associated with Test Enumeration (page X).

How It Works

The Test Automation Framework (page X) uses runtime reflection (or compile time knowledge) to discover all the Test Methods that belong to the test suite and/or all the Test Suite Objects (page X) that belong to a Suite of Suites (see Test Suite Object). It then builds up the Test Suite Objects containing the corresponding Testcase Objects (page X) and other Test Suite Objects in preparation for running all the tests.

When To Use It

We should use Test Discovery whenever our Test Automation Framework supports it. It reduces the effort of automating tests and greatly reduces the possibility of Lost Tests (see Production Bugs on page X). The only time to consider using Test Enumeration is when our framework does not support Test Discovery or when we wish to define a Named Test Suite (page X) that consists of a subset of tests(A Smoke Test[SCM] suite is a good example of this.) chosen from other test suites and the Test Automation Framework does not support Test Selection (page X). It is not uncommon to combine Test Suite Enumeration (see Test Enumeration) with Test Method Discovery; the opposite is less common.

Implementation Notes

Building the Suite of Suites to be executed by the Test Runner involves two things. First, we must find all the Test Methods to be included in each Test Suite Object and second, we must find all the Test Suite Objects to be included in the test run though not necessarily in this order. Each of these steps may be done manually via Test Method Enumeration (see Test Enumeration) and Test Suite Enumeration or automatically via Test Method Discovery and Testcase Class Discovery.

Variation: Testcase Class Discovery

Testcase Class Discovery is the process by which the Test Automation Framework discovers the Testcase Classes on which it should do Test Method Discovery. One solution involves tagging each Testcase Class by subclassing an Testcase Superclass (page X) or implementing a Marker Interface[PJV1]. Another alternative, used in the .Net languages and newer versions of JUnit, is to use a class attribute (e.g. "[Test Fixture]") or annotation (e.g. "@Testcase") to identify each Testcase Class. Another solution is to put all the test case classes into a common directory and point the Test Runner or some other program at this directory. A fourth solution is to use a Testcase Class naming convention and use an external program to find all the files matching the naming pattern. Whichever way we choose to do it, once a Testcase Class has been discovered we can proceed to Test Method Discovery.

Variation: Test Method Discovery

Test Method Discovery involves providing a way for the Test Automation Framework to discover the Test Methods in our Testcase Classes . There are two basic ways to indicate that a method of a Testcase Class is a Test Method. The more traditional way is the use of a Test Method naming convention such as "starts with 'test'". The Test Automation Framework iterates over all the methods of the Testcase Class, selects those that start with the string 'test' (e.g., testCounters) and calls the one-argument constructor to create the Testcase Object for that Test Method. The other alternative, used in the .Net languages and newer versions of JUnit, is to use a method attribute (e.g. "[Test]") or annotation (e.g. "@Test") to identify each Test Method.

Motivating Example

The following code sample illustrates the kind of code that would be required for each Test Method to do Test Method Enumeration if we did not have Test Discovery:

public:
   static CppUnit::Test *suite()
   {
      CppUnit::TestSuite *suite = new CppUnit::TestSuite( "ComplexNumberTest" );
      suite>addTest( new CppUnit::TestCaller<ComplexNumberTest>( "testEquality",
                        &ComplexNumberTest::testEquality ) );
      suite>addTest( new CppUnit::TestCaller<ComplexNumberTest>( "testAddition",
                        &ComplexNumberTest::testAddition ) );
      return suite;
   }
Example ManualTestMethodEnumeration embedded from CPP/CppUnitTestSuitConstruction.cpp

This example is from an earlier version of CppUnit tutorial. Newer versions do not require this any more.

Refactoring Notes

Luckily for us users of existing xUnit family members, the inventors of xUnit realized the importance of Test Discovery and all we have to do is follow their advice on how to identify our test methods. If they implemented it using a naming convention, we may have to do a Rename Method[Fowler] refactoring refactoring to get xUnit to discover our Test Method. If they implemented method attribute, we just add the appropriate attribute to our Test Methods.

Example: Test Method Discovery (using method naming and compiler macro)

When the programming language is capable of managing the tests as objects and invoking the methods but cannot easily find all the methods to use as tests, we may need to give it a small push. Newer versions of CppUnit provide a macro that finds all the Test Methods at compile time and generates the code to build the test suite as illustrated in the previous example. The following code snippet tgriggers the Test Method Discovery:

CPPUNIT_TEST_SUITE_REGISTRATION( FlightManagementFacadeTest );
Example SemiAutomaticTestMethodDiscovery embedded from CPP/CppUnitTestSuitConstruction.cpp

This macro uses a method naming convention to determine what methods ("member functions") to turn into Testcase Objects by wrapping them each with a TestCaller much like in the manual example we saw earlier.

Example: Test Method Discovery (using method naming)

The following examples are notable more for the code that is missing rather than that which is present. Note that there is no code to add the Test Methods to the Test Suite Object.

In this Java example, all the test methods that start with "test" and have no arguments (a total of two) are automatically run by the framework.

public class TimeDisplayTest extends TestCase {
   public void testDisplayCurrentTime_AtMidnight() throws Exception {
      // Setup SUT:
      TimeDisplay theTimeDisplay = new TimeDisplay();
      // Exercise SUT:
      String actualTimeString = theTimeDisplay.getCurrentTimeAsHtmlFragment();
      // Verify outcome:
      String expectedTimeString = "<span class=\"tinyBoldText\">Midnight</span>";
      assertEquals( "Midnight", expectedTimeString,
                    actualTimeString);
   }
  
   public void testDisplayCurrentTime_AtOneMinuteAfterMidnight() throws Exception {
      // Setup SUT:
      TimeDisplay actualTimeDisplay = new TimeDisplay();
      // Exercise SUT:
      String actualTimeString = actualTimeDisplay.getCurrentTimeAsHtmlFragment();
      // Verify outcome:
      String expectedTimeString = "<span class=\"tinyBoldText\">12:01 AM</span>";
      assertEquals( "12:01 AM", expectedTimeString,
                    actualTimeString);
   }
}
Example TestMethodDiscoveryJava embedded from java/com/xunitpatterns/dft/test/TimeDisplayTest.java

Example: Test Method Discovery (using method attributes)

In this C# example, the tests are labeled with the method attribute [Test]. Both CsUnit and NUnit use this way of identifying Test Methods.

   [Test]
   public void testFlightMileage_asKm()
   {
      // setup fixture
      Flight newFlight = new Flight(validFlightNumber);
      newFlight.setMileage(1122);
      // exercise mileage translater
      int actualKilometres = newFlight.getMileageAsKm();   
      int expectedKilometres = 1810;
      // verify results
      Assert.AreEqual( expectedKilometres, actualKilometres);
   }
  
   [Test]
   [ExpectedException(typeof(InvalidArgumentException))]
   public void testSetMileage_invalidInput_attribute()
   {
      // setup fixture
      Flight newFlight = new Flight(validFlightNumber);
      // exercise SUT
      newFlight.setMileage(-1122);
   }
Example TestMethodDiscoveryDotNet embedded from CSharp/NUnitExamples/GoodExamples.cs

Example: Testcase Class Discovery (using class attributes)

Here is an example of using a class attribute to identify a Testcase Class (called a "Fixture" in NUnit) to the Test Runner:

[TestFixture]
public class GoodExamples
{

}
Example TestFixtureDiscovery embedded from CSharp/NUnitExamples/GoodExamples.cs

Example: Testcase Class Discovery (using common location and Testcase Superclass)

The following Ruby example finds all the files with the .rb extension in the "tests" directory and require them from this file. This causes Test::Unit to look for all the tests in each file because the Testcase Class in each file extends Test::Unit::TestCase.

Dir['tests/*.rb'].each do |each|
   require each
end

The Dir['tests/*.rb'] returns a collection of files over which the each do iterates to implement Testcase Class Discovery. The Ruby interpreter and Test::Unit finish the job by doing Test Method Discovery on each required class.

Page generated at Wed Feb 09 16:39:46 +1100 2011

Copyright © 2003-2008 Gerard Meszaros all rights reserved